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ABSTRACT 

The paper describes the development of a course on 
•evaluation in business and industry, offered as part of a Master's 
program in Human Resource Development. Since the average age of the 
students was 36, special emphasis was placed on structuring the class 
to increase relevance to the students' working lives and to ways 
adults learn best. A brief literature review examined theories of 
adult learning and motivation and of experiential learning and 
teaching evaluation. This review resulted in identification of six 
principles of effective practice which guided course design. Central 
to the course's activities was a real-life evaluation study of a 
local insurance company's training programs. Implementation required 
negotiations with the company and university personnel, group design 
of the evaluation study, document analysis, development of employee 
and customer surveys as well as interview questions for individuals 
and focus groups, and preparation of a final report. Finally, the 
course was evaluated against the six principles identified as 
facilitating adult learning: (1) voluntary participation in learning; 
(2) mutual respect; (3) collaborative spirit; (4) action and 
reflection; (5) critical reflection; and (6) self-direction. All 
students recommended using the real-life evaluation project again. An 
appendix presents the evaluation design matrix. 18 references. 
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Introduction 

Considering that the education of adults has been a concern of 
the human race for a very long time, it is curious that there 
has been so little thinking, ' jtigating, and writing about 
adult learning until recently, i'he adult learner has been a 
neglected species. (Knowlcs» 1984, p. 27) 

The above quotation became the premise for a book by Malcolm 
Knowles which began the renewed interest in the process of how adults 
learn and are taught over the last iwo decades. Even more recently, 
however, another author has echoed the same concern: 

The facilitation of learning • assisting adults to make 
sense of and act upon the personal, social, 
occupational, and political environment in which they 
live - is an important, exhilarating, and profound 
activity, both for facilitators and for learners. It is 
also a highly complex psychosocial drama in which 
the personalities in 'olved, the contextual setting for 
the educational transaction, and the prevailing 
political climate crucially affect the nature and form 
of learning. Yet among theorists and practitioners of 
adult learning this complexity is frequently ignored. 
(Brookfield, 1986, p. vii) 

The inspiration to write this paper came from my involvement 
in teaching courses on research and evaluation, instructional design, and 
tiaining and development, to graduate students who have an average age 
of 36. The program in which I teach is geared for individua.13 who 
teach/train adults in business, education, and other non-profit settings. 
The program requires students to take classes on adult learning and adult 
developmental psychology which form the foundation of the remaining 
courses in the program. In teaching my evaluation course for the past four 
years, I have been continually pressed by my students to make the 
content more relevant and concrete. In that my evaluation training done 
at the University of Illinois with Bob Stake, Ernie House, Jim Raths and 
others, has significantly helped me understand and value the theoretical 
underpinnings of evaluation, my students have been looking for 
knowledge and skills they can immediately apply back in their workplace. 
While still Hrmly believing students must understand the various theories 
about evaluation design and implementation, the need to make the course 
more "practical" became a challenge which I took on this year. 
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pefining Adult Learning 



The notion of making knowledge and skills practical and relevant to 
adults is deeply rooted in the adult learning literature which has 
experienced a rebirth of interest in the past twenty years. To clarify what 
the term "adult" means, Knowles (1980) has offered the following two 
criteria: 

1. A person is an adult to the extent that the individual iu 
performing social roles typically assigned by our cultire 
to those it considers adults. 

2. A person is an adult to the extent that the individual 
perceives him or herself to be essentially responsible for his 
or her life. 

Based on the work of Eduard C. Lindeman (1926), who was strongly 
influenced by the work of John Dewey, and. according to Knowles (1984), 
"laid the foundation for a systematic theory about adult learning", he 
offers five key assumptions that constitute the foundation of modern adult 
learning theory: 

1. Adults are motivated to learn as they experience 
needs and interests that learning will satisfy. 

2. Adults' orientation to learning is life-centered. 

3. Experience is the richest source for adults' learning. 
^. Adults have a deep need to be self-directing. 

S. Individual differences among people increase with 
age. 

The implications of these key assumptions are that learning should 
be based on peoples' experiences, should be focused on life situations, that 
instruction should emphasize analyzing experiences, that the role of the 
teacher is that of an "engager" rather than a "transmitter", and that 
teaching strategies should take into account students' differences in style, 
time, place and pace (Knowles, 1984, p. 31). While Knowles has discussed 
the differences between pedagogy and andragogyl at some length, and has 
been criticized for making them sound antithetical, let it suffice to say that 
Knowles (1979) believes that "some pedagogical assumptions are realistic 
for adults and some andragogical assumptions are realistic for childien in 
some situations" (p. 53). 



^ Andragogy may be defmed as the "art and science of helping adults learn". 
(Knowles, 1980, p. 43). 
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Arfii!t«' Mntivfttinn To Learn 

Adults participate in learning for a variety of reasons. Wlodkowski 
(1984) has written that typically adults want "to be better, quicker, and 
more creative in what they value". McLagan (1978) has developed a list of 
items that she believes adults want to gain to some degree- 
Health Security Advancement 
Time Praise (vocational or social) 
Money Comfort Enjoyment 
Popularity Leisure Self-confidence 
Improved Competence Personal prestige 
appearance Efficiency 

Research has also shown, however, that most adults seek out learning 
experiences because of different life-changing events (marriage, divorce, 
new job» promotion, being fired, etc.). Zemke & Zemke (1981) suggest that 
these events are largely produced by the work environment and for that 
reason, learning experiences should be related to their work. For most 
adults in this situation, learning then, "is a means to an end, not an end in 
itself" (p. 58). Given this motivation to learn. Cross (1984) has found that 
older and more mature learners are more likely to select an educational 
program that is less traditional. Furthermore, Cross' research suggests 
that, "the more successful and established the individual is as an adult, the 
less satisfactory traditional college programs are going to be" (p. 77). 

Understanding that adults prefer and seek out learning experiences 
that are motivated by professional needs, choose educational settings that 
offer non-traditional approaches to learning, and come with a life history of 
rich experiences on which tiiey will build new understandings, instructors 
are presented with a challenge to find new or innovative ways that will 
c&pitalize on and stimulate adults' learning. 

Experiential Lear ning and Tcach^ip Evaluation 

"If education is life, then life is also education." 
(Lindeman, 1926, p. 9) 

Several authors have written about the use of a practicum or 
field experience for students of evaluation (Moxley & Visingardi. 1989; 
Morris, 1987, 1989; Brown, 1985, Weeks, 1982) 

Morris (1989) reports having had a great deal of success with 
students engaged in an evaluation course with field experiences. His post- 
course evaluations have asked students to rate how much tiiey feel tlicy 
have learned from doing the evaluation project. On a five-point scale (5- a 
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great deal), students have "responded with an average rating of 4.6". In 
discussing the value with students at the end of the course, he says 
students typically comment "that they 'couldn't imagine' the course being 
offered without the fieldwork component" (1987, p. 8). Moxley & 
Visingardi (1989) add, "The program evaluation practicum is a useful 
vehicle for encouraging students to learn the pragmatic, interpersonal, and 
technical dimensions of evaluation" 'Moxley & Visingardi, 1989, p. S7). 
Through the use of real-life experiences students become in Donald Schon's 
terms, "reflective practitioners" (1983). That is, the student may reflect 

In and on his practice.. .he may reflect on the tacit norms and 
appreciations which underlie a judgment, or on the strategies 
and theories implicit in a pattern of behavior. He may reflect on 
the feeling for a situation which has led him to adopt a 
paiticular course of action, on the way in which he has framed 
the problem he is trying to solve, or on the role he has 
constructed for himself within a larger institutional context" (p. 
62). 

By having students involved in an evaluation project where they are 
responsible for its design and implementation, students are empowered to 
reflect on their action in the way Schon describes. Brookfield (1987) adds, 
"it is through such experience.. that individuals construct the meanings and 
value frameworks that in turn determine how they code new stimuli and 
information" (p. 29). 

Wlodkowski (1986) has written that an instructor should build in 
certain motivational strategies in the learning activity. He suggests that the 
design involve strategies for increasing students' attention, interest and 
involvement: 

1) Attention. "The learner becomes more alert and 
investigates what is occurring in the learning activity." 

2) Interest. "The learner consciously desires more 
information and is emotionally willing to participate in ^ 
the learning activity." Here the learner is listening, 
watching, feeling, reading, and note-taking. 

3) Involvement. "The learner is actively engaged in the 
learning activity. He/she is "searching, evaluating, 
constructing, creating, and organizing the learning 
material into new or better ideas, memories, skills, 
understandings, solutions, or decisions". 



Brookfield (1987, p. 9-10) offers a set of principles that tire 
particularly useful in developing a course based on what is known about 
adult learning. The six principles of effective practice in facili^tating 
learning are: 

1. Participation in learning is voluntary; adults engage 
in learning as a result of their own volition. 

2. Effective practice is characterized by a respect 
among participants for each other's self-worth. 

3. Facilitation is collaborative. Facilitators and 
learners are engaged in a cocperative enterpn.se 
in which, at different times and for different 
purposes, leadership and facilitation roles will be 
assumed by different group members. 

4. Praxis is placed at the heart of effective 
facilitation. Learners and facilitators are involved 
in a continual process of activity, reflection upon 
activity, further reflection and collaborative analysis, 
and so on. 

5. Facilitation aims to foster in adults a spirit of 
critical reflection. Through educational encounters, 
learners come to appreciate that values, beliefs, 
behaviors, and ideologies are culturally transmitted 
and that they are provisional and relative. 

6. The aim of facilitation is the nurturing of self- 
directed, empowered adults. 

These six principles were those that guided the development and 
implementation of my course on evaluation in business and industry this 
past Spring. Additional discussion of these will be offered later in this 
paper. 

ED 514 - Evaluation in Business & Industry - A Case Report 
Background Information 

The class was taught one night a week for fift^n weeks, for 2 1/2 
hours. There were 16 students who were nearing the completion of their 
39 semester credit Master's program in Human Resource Development. 



Prior to the first class I contacted a training and development 
colleague of mine who was working for a mid-size insurance company. I 
asked her if her company had any evaluation needs for which they would 
be seeking outside assistance. It turned out that the company had recently 
made some budget cuts and the evaluation she was hoping to do of the 
company's training programs and their effect on providing quality 
customer service was now an impossibility. She was concerned however, 
because she had included the evaluation as one of her year-end goals. 
When I suggested we meet and talk about my class conducting the 
evaluation, she jumped at the chance 

Negntiations 

About three weeks before the class was officially to begin. Deb (my 
colleague), her boss (Bob, Director of Human Resource Development), and I 
met to discuss their needs. Bob was very interested in our doing an 
evaluation study of the training programs and customer service, but was 
particularly concerned about the confidentiality of information since 16 
"students" he had never seen would be handling some potentially sensitive 
client and employee information. 

Bob suggested that the students develop a charter of confidentiality - 
essentially a statement that would serve as an oath of secrecy about what 
they learned about the company. This seemed like a reasonable request 
and I agreed to it 

After this meeting Bob obtained the President's verbal authorization 
of the project. When we met for the second time, one week before the start 
of class. Bob and Deb were quite excited about the evaluation study. We 
discussed the rationale, purpose and some key evaluation questions as a 
means of focusing the evaluation study. I decided to have as much of this 
information as possible before class so that the students would have a good 
idea of the parameters of the project. I invited Deb and Bob to come to the 
second class meeting to answer studentii* questions and to assist us in 
developing the evaluation plan. 

The Design Phase 

The class appeared very interested in participating in a real 
evaluation study. Most of them weren't sure what it was all about, but 
seemed very willing to continue on this path. They had an infinite number 
of questions which they wrote down in preparation for their interview with 
Deb and Bob. We discussed how to identify the needs of a client, how to 
focus an evaluation study, and the type of relationship we wanted to 
develop with the client. I suggested we use a model alan to Patton's (1982) 
collaborative approach since the evaluation was to be formative. 
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The two-hour interview with Deb and Bob was impressive. It was 
impressive because the students asked many insightful* probing questions 
ranging from why the evaluation was necessary to asking what political 
implications or constraints could possibly impede the evaluation. A great 
deal of information was gained. At the end of this class, the students were 
hooked. They were clearly excited and each one expressed an interest in 
seeing it through to the end. A few days later Bob and Deb both said they 
felt a bit relieved because they could see how professional and committed 
these students were. They now saw them as professional adults rather 
than as "students". 

Deb and I decided to meet weekly so that I could share the progress 
of the study and so that Deb could answer any questions that arose in class. 
By the third class we were developing a design matrix. The purpose still 
seemed a bit fuzzy since we originally thought the study was to evaluate 
how well the training programs were in preparing employees to provide 
"excellent" customer service. But we started to learn that Bob really 
wanted the evaluation to cover customer service in a much broader 
context. He appeared to have another agenda which was not particularly 
clear to us, though I believe he felt the company was not doing what it 
could to provide excellent customer service and saw this as an effective 
means for raising the issue. This perception led us into a discussion about 
the value-laden nature of evaluation and the politics and uses of 
evaluation. While this topic was on the syllabus to be discussed much later 
in the course, the aDpropriate "teachable moment" had arrived, and I seized 
it. 

From here we developed a rationale, purpose statement, list cf key 
audiences, and guiding evaluation questions. When this was presented to 
Deb, she was very pleased with how the class was approaching the study. 
She shared with me, which I later shared with the class, a memo from the 
President which stated that the company had lost significant amounts of 
money in the last three quarters, and as a result, was making some 
additional budget cuts. On this day in my journal I wrote, "Now I know the 
real purpose for this evaluation, and why Bob is so concerned. Though 
neither his nor Deb's position is in jeopardy, the company needs to regain 
its quality status and Bob wants to be part of this movement". 

During the fourth class we talked about different approaches to 
conducting the evaluation, now that we had determined the key evaluation 
questions. The group decided that we were doing a responsive evaluation 
and would use a combination of qualitative and quantitative measures. Wc 
then developed a data collection matrix of evaluation questions and 
methodology (see Appendix A). It was decided that M address the nine 
evaluation questions, the following methods would be used: 
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• employee focus group interviews (3) 

• manager focus group interviews (2) 

• customer phone interviews (N=70) 

• former customer phone interviews (N=2S) 

• non-policy claimant phone interviews (Ns 20) 

• individual manager interviews (Ns9) 

• regional and district sales manager phone 
interviews (N= 25) 

• employee survey (N= 122) 

• current customer survey (N= 588) 

• document analysis 

It should be noted here that we were prevented from contacting and 
involving the insurance agents. At the beginning of our negotiations, Bob 
told me that we were not IG survey or talk with the Held agents. He 
explained that this group had been surveyed the summer before and would 
not be receptive to participating in this study. He indicated that the politics 
of the organization would not support gathering data from this group. This 
fact not only troubled me, but more interestingly, it deeply bothered the 
students. If this study was about customer service, they said, how could 
they not involve the agents who are the first line providers of customer 
service. We brought this up several times to Bob, but he stood firm. We 
discussed this issue in class and concluded that we would clearly state this 
as an evaluation constraint in our final report that limits the findings of the 
study. 

Class #5 involved a discussion about sampling, reliability and 
validity. The students appeared to really understand these topics because 
they could immediately relate their meaning to a concrete example in 
which they were becoming closely involved. We decided that the work 
would be divided up among all the students. Since almost all of the 
students were employed full-time, I asked if anyone would be able to 
conduct the focus groups (we agreed that each group would have two 
facilitators). To my surprise, twelve of the sixteen said they very much 
wanted to be part of this activity. For several, this meant taking vacation 
or sick days from work. This was a clear sign to me that they were 
invested. 

Another example of student interest and commitment occurred when 
one student suggested that we find out more about customer service and 
volunteered to bring in a series of videotapes on this topic. I told the class 
that they could view these tapes one-hour prior to class if they were 
interested. For tive weeks, 6 or 7 people came early to view these tapes. 
These particular tapes were useful in identifying some of the issues 
company's must pay attention to in order to provide quality customer 
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service. After the fifth class, I wrote in my journal, "they are really in this 
together - it feels so right." 

Implementation 

Before the next class, two of the students and I met with some of the 
key managers, both to get information and to solicit their support. The 
students were surprised at how helpful and informative these people were. 
During the next class, we discussed this observatio n, and talked about 
peoples' perceptions of evaluation and the role of the evaluator. 

The topic for class #6 was document analysis. We had a pile of 
documents from the company; everything from the annual report, to recent 
market research surveys, to telephone audit reports. After a lecture on 
document and record analysis, the students were able to apply what they 
learned and analyzed the documents we had collected. 

Following the document analysis we talked about survey design. 
Following this lecture/discussion, I divided the students into work groups 
of four and they started developing questions and response formats for the 
customer survey. For the employee survey, I asked everyone to bring to 
class next time three questions to be considered. After each group came up 
with a set of questions (keyed to the guiding evaluation questions), they 
exchanged these with another group who reviewed and critiqued the 
questions. After this class I took the questions for the customer survey 
and constructed a draft of the survey. 

Even though I teach students always to expect the unexpected, and 
to even plan for this, this project was thrown a curve on class #7. We had a 
terrific snowstorm that shut the Twin Cities down for 24 hours. We had 
counted on this class period to develop the employee survey and to begin 
developing the phone interview guides. I wrote in my journal, 

"How frustrating - so much to do and even less time to do it in. 
Already we're behind and class #8 is coming up. One thing I've 
learned already is that this project, while exciting as a learning 
experience, is far more complex than I expected it to be. It is 
turning out to be a lot of work for students and me. The 
balance between teaching and student involvement is so hard 
to attain." 

During class #8 we spent time debriefing the findings of the 
document analysis and decided that several pieces of information would be 
useful in developing the instruments. We then worked in small groups 
developing the survey and interview instruments. Having them develop 
these tools after listening to a lecture on instrument construction was very 
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useful. They could apply this infonnation immediately and receive 
feedback on how well they understood it. 

During the small group work I met with each person individually and 
asked which of the data collection tasks they were most interested in 
working on. Silently worried that I would end up doing most of the work, I 
again was amazed how willing everyone was to do several activities. 

Deb had agreed to do all of the scheduling for the individual and 
focus group interviews. This made our task substantially easier. I gave 
her dates when the students were available and she worked with the 
company's employees in confirming these dates. Deb also agreed to 
provide mailing labels and envelopes for all of the customer surveys. We 
would provide the return address envelopes so that they surveys could be 
returned to the College.^ Deb and Bob also said that they had sent out 
memos to the departments that would be involved in the evaluation, 
explaining the purpose and intent of the study. 

For class #9, we spent the majority of time finalizing drafts of the 
employee and customer surveys and wrote cover letters for each. The 
following day I delivered them to Bob and Deb. They reviewed them, made 
some suggestions which I incorporated, and I returned the master copy to 
Deb to have typeset* reproduced and mailed. 

The topic of class #10 was focus group and individual interviewing. 
After a lecture on each of these the students developed the interview 
guides and engaged in a series of mock interviews. We discussed various 
aspects of interviewing and role-played some possible responses. The 
students were excited about the focus groups which were scheduled for the 
following week. Nine students would be facilitating the focus groups and 
we agreed that I would sit in on each of them for a about 45 minutes. We 
also agreed that we would meet at the end of their particular sessions and 
debrief 'their experience.^ 

The focus groups went like clock-work. The students all arrived at 
least 30 minutes early, and were w^U-versed with their questions. They 
were more than ready when the participants started arriving. I observed 
each of the students and later provided them with written and verbal 
feedback. The participants in the focus groups cooperated very well. Some 



2 We chose this method for two reasons: 1) to provide confidentiality to the 
respondents, and 2) to decrease the amount of instrument handling. 

3 One of the course assignments was a thought paper that was to be a personal 
reflection about their role in the evaluation study. It was to include what they 
did, how they felt about it, what they wished they had known before, and what 
they would do differently next time. 
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knew about the study, others did not. The focus groups revealed a 
situation for which we weren't quite prepared, however. We learned that 
employee morale was quite low; that the climate of the organization was 
seriously troubled. The employees knew that the company was having 
Hnancial problems, and many said they didn't know if they had a future 
with the company. Many looked anxious and spoke angrily. Some said 
they knew of many who were looking for other jobs. In spite of feeling a 
certain loyalty to the company, several explained that recently, their 
service to the customer might have been less than excellent since they 
have been so concerned about losing their jobs. Considering this turn of 
events, and the climate it developed in the focus groups, I believe that the 
students did remarkably well. Their thought papers were insightful and 
were proof that the experience was full of learning. 

Class #11, being the day of the focus groups gave us much to talk 
about. The focus group facilitators shared their experiences with the ^cst 
of the class. 

ik few days later I was scheduled to interview the President and 
Vice-President of the company. When I met with Bob briefly before these 
interviews, I sensed that Bob was upset or nervous about something he 
would not share. Bob wanted me to promise that 1 would not share any of 
the preliminary results with the President. I agreed, saying that that 
would be my policy in any case. Upon returning to Bob's office after these 
interviews, I found Bob visibly upset. Deb locking shaken, and one of her 
colleagues crying. What I learned was that while I was away interviewing, 
Deb and one of her co-workers had been told they were being laid off 
because of severe problems the company was having, both financially and 
with their credit rating. Others in the company were also being let go too. 
Rumors about merger, sell-out, and bankruptcy were abounding. The 
future of the company was unclear. 

In spite of all this. Bob said he wanted to continue with the 
evaluation. He said, "it will be especially interesting now". Due to delays in 
duplicating the survey and sign-offs on the cover letters, the customer and 
employee surveys were going out the next day. We had only four more 
''lass sessions. 

Class #12. When I told the class about what had happened with Deb, 
they were shocked and quite upset. They asked questions about what does 
this mean for the evaluation? One person even said, "can we change the 
focus of our evaluation and study the company's culture and climate 
instead?" The students agreed that they had become involved in a very 
interesting project, one that definitely reflected "real life". For the rest of 
the class period we spent time talking about content analysis. 
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I gave the students transcripts for each of the focus groups (they had been 
audio-taped). They worked in groups using what they had learned about 
content analysis coding the data. After they spent time on this activity, we 
discussed the issues that arose in analyzing the ddta and the decision rules 
that guided each group's analysis. 

A lecture on quantitative data analysis was the topic of class #13. 
We looked at the surveys, noted the level of data it would gather, and 
discussed appropriate analysis techniques. With the surveys that had been 
returned, the class learned about cleaning surveys and proceeded to code 
them for computer entry. 

The topics for class #14 were the evaluation standards and 
utilization. After a mini-lecture on the standards we talked about how our 
evaluation conduct met tht standards and what we might have done 
differently. After my lecture on evaluation utilization, I asked the class to 
think about the factors they thought might influence this evaluation's use 
and the factors that might influence evaluation use in their organizations. 

It was the last class (#15), and we hadn't even received the print- 
outs from the computing center with the results of the employee and 
customer surveys. We had individual interviews that had been transcribed 
from audio-tape» but had not had time to analyze them. Some students 
approached me and said that they really wanted to learn how to do the 
remaining steps of the analysis and suggested we somehow get together to 
work on this. With a sense of relief, I set a date, reserved a room, and told 
the class that if they wanted to come, and could come, we would analyze as 
much of the data as possible in one afternoon. It wouldn't all get done, but 
it would provide them with more concrete practice on how to analyze both 
survey and interview data. Five people were able to come this day. We 
did not do everything, but accomplished a great deal. The semester had 
been over for two weeks, and I was left with the task of analyzing the 
remaining data and writing the report. I was disappointed that I wasn't 
able to give students practice doing this, since I believe it is so obviously 
critical to any evaluation study. Also, the report was due in two weeks. 

The Report 

I called Bob and told him that the project was a little behind 
schedule but that the report would be forthcoming. He reiterated his 
interest, said morale was terrible, and said he wasn't sure how long he was 
going to be around. About two weeks later. Bob called and again asked 
how I was doing on the report. I told him it was coming along and I would 
have it to him in two more weeks. He surprised me when he said, that the 
evaluation study had recently taken on new meaning with some. He 
explained that the Vice-President (to whom Bob reports) wants to "use it as 
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a scimitar" to wave over his head. In further discussions I was led to 
believe that Bob possibly saw the evaluation as his legacy, a kind of "I told 
you so, now see for yourself kind of symbol. I only \(dshtd my students 
were around so we could have discussed this new development. 

I Hnally delivered one copy of the report to Bob and asked him to 
review it. A few days later he called and said he liked it and had just a 
couple of questions. We discussed these and I made some minor 
ad.'ustments. I then provided him with the seven copies he had requested. 
He gave three to his boss, who then gave one to the President (who had 
recently resigned, or had been forced out, but was staying on until a 
successor could be named), and one to the ex-President who had been 
called on to help the company get through its crisis. I told Bob that I would 
be available to make any verbal presentations requested about the study. 
After several weeks, I called Bob, and asked him what was new, and if 
anyone had read and responded to the evaluation report. He said, that the 
company was standing still, "people are treading water". For one reason or 
another, the Vice-President and Presidents are "sitting on it" and he wasn't 
sure why. He explained that he had recently been working on his budget 
for next year, and had requested signified itly more money in some areas. 
He said he was justifying these projected expens. s based on the evaluation 
results. He thinks that when the Finance department sees his requests and 
his references to the evaluation study, it will prompt management to ask 
questions about the evaluation, which Bob hopes will be the start of 
discussions around the Hndings of the study. 

Did the Course Meet Brookfield's (1987^ Six Principles? 

As one method of evaluating this approach to teaching, I will discuss 
how well my evaluation course achieved each of Brookfield's six principles 
of effective practice. 

1. Participation in learning is voluntary. 

Brookfield suggests that once learners have expressed a desire to 
learn, and have enrolled in some type of educational experience, the 
facilitator/instructor will not have to spend a great deal of time 
trying to coerce students to engage in learning activities. He 
explains, "Adult's willingness to learn also means that they are less 
likely to resist participatory learning techniques such as discussion, 
role playing games, small-group work, and collaborative analysis of 
personal experiences" (p. 11). At the same time, learners may 
withdraw from any activity which does not meet their specific needs 
or does not make particular sense to them. 



The project in which my class was involved allowed them to 
participate in varying degrees according to their needs and interests. 
While they were required to write three thought papers and 
complete the development of a final project (an evaluation plan), 
they were not required to conduct the focus groups, telephone 
interviews or personal interviews for their grade. Yet several 
members of the class sacrificed vacation time and met after the end 
of the semester to complete this project. There was a high degree of 
voluntary behavior because they could see the benefit of "real life" 
experiences. For the majority of students, they anticipated being 
involved in evaluation activities in their place of employment and 
viewed the class more as a laboratory where they could practice and 
hone their evaluation skills. In this respect, I believe the class 
project met Principle #1. 

Mutual Respect 

This principle refers to the need individuals have to feel "valued as 
separate, unique individuals deserving of respect" (p. 13). Brookfield 
adds, "Successful educational experiences are generally characterized 
by the evolution of some form of group consensus regarding 
acceptable behaviors by participants" (p. 13). This responsibility is 
both the learners' and the instructor's. It is up to the instructor to 
provide a climate in which people can feel free to challenge ideas and 
at the same time, take risks. 

Though not planned initially, but implemented because of the client's 
request, the confidentiality charter was the first step in bringing 
about this atmosphere of mutual respect. When I ^rst mentioned 
this request to the students, thcu reaction was not particularly 
positive. Some saw it as silly and others were resentful because they 
interpreted it as not being trusted by the client. After much 
discussion, however, the group came together and agreed to develop 
a charter that was simple and straightforward. It took them 
approximately S minutes to develop this once they had aired their 
thoughts and understood the issues involved. The respect they 
developed was not only for the client and their needs, but for the 
individuals in the class as well. 

As much as possible I tried to support this climate of respect. 
This meant allowing and even encouraging the group to challenge 
one another and me about anything that happened during the 
project's implementation. The fact that the students remained so 
involved, and raised so many relevant, difHcult and meaningful 
questions, and strongly recommended this approach be continued in 
future classes, suggests that this principle was achieved. 



Collaborative Spirit. 



One of the differences between school education and adult education* 
according to Broolcfield, is the existence of some kind of participatory 
and collaborative element in the educational process for adults. This 
collaborative approach to learning means that the learners share in 
the responsibility for "posing questions, identifying materials, 
suggesting priorities, and organizing aspects of the group process" (p. 
14). 

I believe this principle was achieved at various stages throughout 
the evaluation project. In the beginning, students developed and 
retined the key evaluation questions. They made decisions about the 
appropriate data collection strategies. They participated in work 
groups to develop and critique the data collection instruments. In 
addition, they worked together in analyzing and interpreting the 
open-ended and focus group data. This project depended upon their 
experiences, new learnings, and willingness to participate and be 
collaborative. 

Action and Reflection. 

According to Brookfield, "This process, centers on the need for 
educational activity to engage the learner in a continuous and 
alternating process of investigation and exploration, followed by 
action grounded in this exploration, followed by reflection on this 
action, followed by further investigation and exploration... and so on" 
(p. IS). The way to do this Brookfield suggests, is through providing 
people with skills and having them apply them in "real life settings". 

Clearly this principle was achieved through the reported project. At 
each phase of the evaluation, or syllabi topic, the class discussed 
what their experiences had been, what they thought would happen, 
what did happen, and what it meant for future practice. They 
continually reflected on the various topics and eventually began 
constructing new ways of seeing the issues presented to us. Through 
this reflective practice based on action, the students grew to 
understand evaluation as a Held of study and practice. 

Critical Reflection. 

Brooktield takes the concept of reflective practice an additional step 
farther He notes the difference between education and training 
where training is the transmittal of a set of identified skills. 
However, "in education...leamers are encouraged to examine the 
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assumptions ur srlying the acquisition of skills, to consider 
alternative purposes, and to place skill acquisition in some broader 
context" (p. 17). The key point here, according to Brookfield, is that 
"education is centrally concerned with the development of a critically 
aware frame of mind, not with the uncritical assimilation of 
previously deHned skills or bodies of knowledge" (p. 17). 

This principle was probably best enacted during discussions about 
methodology and the epistemology underlying quantitative m'\ 
quantitative approaches to evaluation. Many of the students had not 
thought about alternative ways of collecting data nor the 
assumptions underlying positivism or phenomenology. These 
discussions were lively and generated several new insights for the 
students. The inclusion of this principle helps students understand 
the need for understanding practice through learning the theoretical 
foundations of disciplined inquiry. 

6. Self-Direction. 

Brookfield defines self-directed learning as, "The adult's assumption 
of control over setting educational goals and generating personally 
meaningful evaluative criteria." He adds that self-directed learning 
is, "A matter of learning how to change our perspectives, shift our 
paradigms, and replace one way of interpreting the world with 
another" (p. 19). 

I think this principle is the most difficult to obtain. The students 
were not quite as independent as Brookfield and other theorists on 
adult learning might have wished. Instead of developing individual 
learning contracts that are often suggested as a way of encouraging 
self-directed learning, students were more or less guided by my 
conception of what needed to be accomplished in the evaluation 
study. At the same time, they did learn new ways of thinking about 
evaluation and its practice. The project being acted out in "real life" 
did challenge the students' to think in new paradigms. This, of 
course, was achieved to varying degrees with the students. 

Post Hoc Observations about Experiential Evaluation Projects 

Aside from the traditional end-of-course evaluation required by my 
department and college, I asked the students to respond to a four question 
survey that focused only on the evaluation project. Without exception, 
students recommended using this approach again. 

I have learned several things, however, that I will do differently 
next time. For one, the scope of the project was much too large. It seemed 
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to grow out of proportion the longer we were engaged in it. Next time, I 
will make sure that we not take on more than the students can reasonably 
accomplish in one semester. As Morris (1987) suggests, small scale 
evaluations have a much greater chance of being completed within a 
semester framework. He believes that Master's level students, "gain more 
fully carrying out an evaluation of limited scope than from being involved 
in a subset of the phases associated with more ambitious projects" (p. 10). 

When students were asked how they would improve the process of 
teaching this couise, several suggested that ways to allow students greater 
panicipation in the data analysis and report writing be developed. Not 
only is this beneficial for the student, but it relieves the instructor of a 
great responsibility. 

In the beginning of this paper I spoke about the conflict I've had in 
finding the right balance between theory and practice for teaching 
evaluation. In teaching this class using an experiential method, I found it 
extremely difficult to find the time to address theoretical issues as deeply 
as I would have liked. While we did not neglect theoretical issues entirely, 
two students mentioned on their evaluation forms, that they felt they 
might have missed some theory because of the amount of time we spent 
"practicing" evaluation skills. Yet, these same students recommended using 
this approach the next time the course is taught. While I have not resolved 
the theory vs. practice conflict, I do feel that the experiential nature of this 
course, based on adult learning principles, provided the learners with a 
body of knowledge and a set of skills that will greatly benefit them in both 
their professional and personal lives. 
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APPENDIX A 
EVALUATION DESIGN MATRIX 

DATA SOURCES 



KEYQUESTION 



CUSTOMER 



NON-POLICY 
CLAIMANTS 



R.M. & D.M. 



MSI 

PORTER MANAGE- 
EMPLOYEES DCX:UI\>'EhnS CUSTOMERS MENT 



1. What organizational factors affect Survey 
employees' ability to provide high 

quality customer service? 

2. How do vert)al and written communication Survey 
processes affect the quality of customer 

service? 

3. How effective have the training initiatives been 
in Increasing employees' ability to provide 
quality customer service? 



4. What additional training would liltely increase 
the level of customer service provided 

by employees? 

5. How do employees perceive the goals of 
oroviding customer service? 

6. What do employees perceive to be the obstacles 
in providing quality customer service? 

7. How consistent is the service being provided Survey 
by employees and agents? 

8. How satisfied are customers with the level Survey 
of customer service provided by? 

9. What recommendations do customers and Survey 
employees have for improving the level 

of quality service provided by? 



ERIC 
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Phone Int. Phone int. Focus Group X 

Survey 



Phone int. Focus Group 
Survey 



Phone int. 



Phone int. 

Phone int. 

Phone int. 

Phone int. 
Phone Int. Phone int. 
Phone Ine. Phone int. 



"Call Home- 
Observation 
Focus Group 
Survey 

Focus Group 

Survey 

Observation 

Focus Group 
Syivey 

Focus Gtoup 
Survey 

Focus Group 
Survey 

Focus Group 
Survey 

Focus Group 
Survey 



X 
X 
X 
X 
X 



Phone InL Pers. 

Interview 



Phone Int. Pers. 

Interview 



Pers. 
Interview 



Pers. 
interview 



Pers. 
Interview 

Pers. 
Interview 

Phone Int. Pers. 

Interview 

Phone Int. Pers. 

Interview 

Phone Int. Pers. 

Interview 
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